Fast parameter estimation for joint maximum entropy language models
نویسنده
چکیده
This paper discusses efficient parameter estimation methods for joint (unconditional) maximum entropy language models such as whole-sentence models. Such models are a sound framework for formalizing arbitrary linguistic knowledge in a consistent manner. It has been shown that general-purpose gradient-based optimization methods are among the most efficient algorithms for estimating parameters of maximum entropy models for several domains in natural language processing. This paper applies gradient methods to whole-sentence language models and other domains whose sample spaces are infinite or practically innumerable and require simulation. It also presents Open Source software for easily fitting and testing joint maximum entropy models.
منابع مشابه
Collaborative Filtering Efficiently Using Purchase Orders
We propose a new collaborative filtering method that can predict the next purchase item by efficiently using the sequential information in purchase histories for recommendations. Markov models and maximum entropy models are both widely used techniques for such recommendations. In Markov models, parameters can be estimated and updated fast and efficiently, but predictions may not be accurate. On...
متن کاملMaximum Entropy Modeling Toolkit
The Maximum Entropy Modeling Toolkit supports parameter estimation and prediction for statistical language models in the maximum entropy framework. The maximum entropy framework provides a constructive method for obtaining the unique conditional distribution p*(y|x) that satisfies a set of linear constraints and maximizes the conditional entropy H(p|f) with respect to the empirical distribution...
متن کاملMildly context sensitive grammars for estimating maximum entropy parsing models
The maximum-entropy framework provides great flexibility in specifying what features a model may take into account, making it effective for a wide range of natural language processing tasks. But because parameter estimation in this framework involves computations over the whole space of possible labelings, it is unwieldy for the parsing problem, where this space is very large. Researchers have ...
متن کاملEfficient estimation of maximum entropy language models with n-gram features: an SRILM extension
We present an extension to the SRILM toolkit for training maximum entropy language models with N -gram features. The extension uses a hierarchical parameter estimation procedure [1] for making the training time and memory consumption feasible for moderately large training data (hundreds of millions of words). Experiments on two speech recognition tasks indicate that the models trained with our ...
متن کاملA Comparison of Algorithms for Maximum Entropy Parameter Estimation
Conditional maximum entropy (ME) models provide a general purpose machine learning technique which has been successfully applied to fields as diverse as computer vision and econometrics, and which is used for a wide variety of classification problems in natural language processing. However, the flexibility of ME models is not without cost. While parameter estimation for ME models is conceptuall...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2004